Multitouch data fusion

ABSTRACT

A method for performing multi-touch (MT) data fusion is disclosed in which multiple touch inputs occurring at about the same time are received to generating first touch data. Secondary sense data can then be combined with the first touch data to perform operations on an electronic device. The first touch data and the secondary sense data can be time-aligned and interpreted in a time-coherent manner. The first touch data can be refined in accordance with the secondary sense data, or alternatively, the secondary sense data can be interpreted in accordance with the first touch data. Additionally, the first touch data and the secondary sense data can be combined to create a new command.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.11/963,516, filed Dec. 21, 2007 and published on Sep. 4, 2008 as U.S.Publication No. 2008-0211766, which claims the benefit of U.S.Provisional Application No. 60/879,152, filed Jan. 7, 2007, the contentsof which are incorporated herein by reference in their entirety for allpurposes.

FIELD OF THE INVENTION

This relates to systems utilizing multi-touch sensitive input devicesand other input devices, and more particularly, to the combining ofmulti-touch input data with data from other input devices to gain anadvantage thereby increasing the efficiency and performance of inputtingoperations.

BACKGROUND OF THE INVENTION

Systems may have multiple input means. However, each input means istypically operated independently of each other in a non seamless way.There is no synergy between them. They do not work together or cooperatefor a common goal such as improving the input experience.

SUMMARY OF THE INVENTION

While the fingertip chording and movement data generated by multi-touchinput devices can provide a strong set of user control means, additionalinformation from other sensing modalities when combined or fused withthe chording and movement data can significantly enhance theinterpretative abilities of the electronic device and/or significantlyimprove the ease of use as well as streamline input operations for theuser. Therefore, embodiments of the invention propose the concept of MTdata fusion, which is defined as the combination of data from one ormore independent sensing modalities with chording and movement data froma MT sensor in order to improve the operation and use of an electronicdevice.

There are a number of independent sensing modalities that when fusedwith Multi-touch chording and movement data provide enhanced performanceand use of electronic devices. The sources of independent sensing datafall into several categories: (1) those that measure some aspect of theuser's body state, (2) those that measure data from the environment,which could include sensing data from other individuals, and (3) thosethat measure some aspect of the state of the electronic device.

In accordance with one embodiment, one or more of these independent datasources can be fused temporally with movement and chording data from amulti-touch sensor to significantly enhance the performance and use ofelectronic devices. The information flowing from the various sources canbe combined or fused such that events in each data stream are timealigned with each other. As such, the multiple data streams can beproperly understood in conjunction with the other.

In accordance with other embodiments, the results of voice recognitionand speech understanding can be fused with multi-touch movement data insuch a way as to significantly enhance electronic device performance.The contact size and contact separation of touch data along with fingeridentification data (such as from a camera) can allow the multi-touchsystem to make guesses concerning finger identification of the touchdata. Gaze vector data (the determination of a user's gaze) can be fusedwith touch data and/or objects appearing on a display to perform variousoperations such as object movement or selection. The fusion of devicedynamics data (e.g. movement data) with multi-touch movement data canresult in a smoothing out (i.e., improved filtering) of unintendedfinger motion due to the means of traveling (e.g., vibrations andjolts).

Biometric inputs include, but are limited to, hand size, fingerprintinput, body temperature, heart rate, skin impedance, and pupil size.Typical applications that might benefit from the fusion of biometricdata with multi-touch movement data would include games, security, andfitness related activities. Facial expressions conveying emotional statecan also be fused advantageously with multi-touch movement data duringcreative activities such as music composition.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram of an electronic device or system utilizingmulti-touch (MT) data fusion in accordance with one embodiment of thepresent invention.

FIG. 2 is a MT data fusion method including MT sensing and secondarysensing in accordance with one embodiment of the present invention.

FIG. 3 is a MT data fusion method including the generation of MT andsecondary data streams in accordance with one embodiment of the presentinvention.

FIG. 4 is a method of inputting including the collection and combiningof MT and secondary sensing data in a time-coherent manner in accordancewith one embodiment of the present invention.

FIG. 5 is a method of inputting wherein the secondary sensing data isutilized to supplement the MT data in a time-coherent manner accordancewith one embodiment of the present invention.

FIG. 6 is a method of inputting wherein the secondary sensing data isutilized to interpret the MT data in a time-coherent manner inaccordance with one embodiment of the present invention.

FIG. 7 is a method of inputting including voice recognition thattriggers actions associated with chording and movement data inaccordance with one embodiment of the present invention.

FIG. 8 is a method of inputting wherein a voice modify command triggersactions on an object in accordance with one embodiment of the presentinvention.

FIGS. 9A-9E are diagrammatic illustrations of fusing voice data with MTdata according to one embodiment of the invention.

FIG. 10 is illustrates the fusion of voice recognition and MT operationsaccording to embodiments of the invention.

FIG. 11 is a method for unambiguous finger identification in accordancewith one embodiment of the present invention.

FIG. 12 is another method for unambiguous finger identification, inaccordance with one embodiment of the present invention.

FIG. 13 is an inputting method including the matching of fingers tocontacts in accordance with one embodiment of the present invention.

FIG. 14A is an illustration of an exemplary image of a pair of handsthat are positioned over a MT surface, and FIG. 14B is an illustrationof an exemplary image of an arrangement of contacts at the MT surface.

FIGS. 15A and 15B are diagrams of an electronic device including a MTsurface and an imaging device in accordance with one embodiment of thepresent invention.

FIG. 16 is a diagram of an electronic device including a MT surface andan image sensor in accordance with another embodiment of the presentinvention.

FIG. 17 illustrates an image sensor located in a display or all in onecomputer 340 in accordance with another embodiment of the presentinvention.

FIG. 18 is an inputting method including the fusion of a gaze vectorwith MT sensing in accordance with one embodiment of the presentinvention.

FIG. 19 is an inputting method including taking action on an objectbased on a gaze in accordance with one embodiment of the presentinvention.

FIG. 20 is an inputting method including the filtering of MT data basedon motion data in accordance with one embodiment of the presentinvention.

FIG. 21 is an operational method wherein MT data can be interpreteddifferently depending on orientation data in accordance with oneembodiment of the present invention.

FIG. 22 is an inputting method including biometric sensing in accordancewith one embodiment of the present invention.

FIG. 23 is an operational method including collecting emotional data andtaking actions based on the emotional data in accordance with oneembodiment of the present invention.

FIG. 24 is an inputting method including force sensing in accordancewith one embodiment of the present invention.

FIG. 25 is an inputting method including capturing and comparing MT datafrom different devices in accordance with one embodiment of the presentinvention.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT

Embodiments of the present invention propose combining or fusing multitouch sensing with other sensing modalities to gain an advantage therebyincreasing the efficiency and performance of inputting operations.

In Multi-touch 2D sensing, as used as the primary or secondary means toallow a user to control the function and operation of an electronicdevice, the positions of all finger tips in contact or close proximityto a sensing surface are tracked and recorded. The arrangement ofcontacts (e.g., chords) and the movement of the contacts (e.g.,gestures) at or near the sensing surface are interpreted by theelectronic device as commands from the user meant to modify, initiate,or terminate a function performed by the electronic device.

While the fingertip chording and movement data provides a strong set ofuser control means, additional information from other sensing modalitieswhen combined or fused with the chording and movement data couldsignificantly enhance the interpretative abilities of the electronicdevice and/or significantly improve the ease of use as well asstreamline input operations for the user. Therefore, embodiments of theinvention propose the concept of MT data fusion, which is defined as thecombination of data from one or more independent sensing modalities withchording and movement data from a MT sensor in order to improve theoperation and use of an electronic device.

There are a number of independent sensing modalities that when fusedwith Multi-touch chording and movement data provide enhanced performanceand use of electronic devices. The sources of independent sensing datafall into several categories: (1) those that measure some aspect of theuser's body state, (2) those that measure data from the environment,which could include sensing data from other individuals, and (3) thosethat measure some aspect of the state of the electronic device. By wayof example, the sense data may include, but are not limited to, thefusion of voice, finger identification, gaze vector, facial expression,hand-held device dynamics, and biometrics such as body temperature,heart rate, skin impedance, and pupil size. It should be noted thatembodiments of this invention are not directed at individual sensingmeans. They are directed, instead, at the temporal fusion of data fromexisting sensing means with multi-touch movement and chording data toenhance electronic device ease of use and performance.

In accordance with one embodiment, one or more of these independent datasources can be fused temporally with movement and chording data from amulti-touch sensor to significantly enhance the performance and use ofelectronic devices. Generally, temporal fusion signifies an apparentcoming together of several events in time (multiple sets of data) withina single individual context (computer application, mode or platform).More specifically, in temporal fusion, the data from multiple sources(MT+other sensing modality) is interpreted in a time coherent manner.The information flowing from the various sources are combined or fusedsuch that events in each data stream are time aligned with each other.As such, the multiple data streams can be properly understood inconjunction with the other.

Some examples of MT data fusion are given below.

Voice Fusion

Voice input, speech recognition, and language understanding all fallunder the long-sought goal of enabling electronic devices to performtheir intended function directed by human speech. In recent years, muchprogress has been made in enabling the recognition of speech byelectronic devices. Language understanding, being much more difficult,has not enjoyed the same level of success. Embodiments of the inventiondescribed herein are not concerned with how voice data is recognized orunderstood. Embodiments of this invention simply make use of the resultsof voice recognition and, eventually, speech understanding asindependent sensing inputs to be fused with multi-touch movement data insuch a way as to significantly enhance electronic device performance.

There are a large number of examples where voice and multi-touch can befused to add significant benefit to an application. The most benefit isgained when voice and multi-touch use is partitioned where they aremaximally effective. In other words, voice input is applied to actionspoorly served by manual input and manual input handles tasks poorlyserved by voice. For example, mode selection or static commands are moreefficiently done using voice input while moving objects on the screen isbest done manually with MT. One example will now be given. In theediting of, say, a mechanical drawing the task may be to select andmodify the objects making up the drawing. Assume the modification ofeach object involves resizing, rotation, and color change. Furtherassume that the task is not a simple scaling or rotational change ofeach object. The minimum effort, therefore, is expended when theresizing and rotation is done by using multi-touch gestures (i.e.,manually) and when the color change is done using voice input. Considerthe alternative: Using voice to resize and rotate each object isproblematic because a verbal description of the intended size androtation is difficult to express. Using multi-touch to select a color istypically less efficient than using voice because the color has to beselected by traversing a list. Alternatively or additionally, voiceinput may be used to insert text in the object.

Finger Identification Fusion

Finger identification means that the fingers currently touching or inclose proximity to the multi-touch sensing surface are identifiedwithout ambiguity as to their names and the hand (i.e., right, left,owner) they belong to. For example, let's assume the index and middlefingers of the right hand are in contact with a multi-touch surface.Most, if not all, multi-touch sensors cannot unambiguously classify thecontacting fingers as index and middle from the right hand. Usingcontact size and contact separation allows the multi-touch system tomake guesses concerning finger identification but the accuracy of theguess is typically not good unless sufficient number of fingers from onehand are in contact with the surface.

The source of independent finger identification data is readilyavailable from a camera such as an over-the-multi-touch surface camera.The camera data shows where the fingers of each hand are relative to themulti-touch XY coordinates. The camera cannot necessarily determine ifthe fingers are in contact with the touch surface but this is notimportant since the fused data from the camera and the multi-touchsensor will provide unambiguous finger movement data that includes thevertical (i.e., Z dimension) position of each finger. A typicalembodiment of an over-the-multi-touch-surface camera for a notebookcomputer could be one or more embedded iSight cameras each with a swingmirror that would enable imaging both hands over a multi-touch surface.

Finger painting, where each finger has an assigned color, stroke, orother characteristic, is a simple example of an application that wouldbe significantly enhanced compared to the state-of-the-art by usingfinger identification with multi-touch data fusion. Without unambiguousfinger identification, the application, whether it is finger painting orsomething else, would not be able to maintain the proper assignment ofparticular attributes given to a specific finger. For example, if theindex finger of the left hand is assigned the color red and the otherfingers are assigned different colors the application must be able todetermine when the index finger of the left hand is in contact with thesurface in order to paint red. Conversely, the application must be ableto determine when the red-assigned finger is not in contact with thesurface. The fusion of finger identification data with multi-touchmovement data allows the application to function without error.

Gaze Vector Fusion

Over the last twenty years there has been a fair amount of research anddevelopment in gaze directed user interfaces. Most of the effort hasbeen focused on providing computer interface solutions to people withdisabilities who are not able to use a keyboard or mouse. Research hasalso been done on using gaze direction in virtual reality applications.As with the other sensing modalities discussed in this disclosure thetemporal fusion of gaze direction data with multi-touch movement datacan be used to enhance the performance of electronic devices. Therefore,the capture of gaze vector data and the methods for computing gazedirection will not be discussed herein.

There are many possible applications that would benefit from thetemporal fusion of gaze vectors with multi-touch movement data. For thepurpose of example, one simple application will be discussed here:Consider a typical computer screen, which has several windows displayed.Assume that the user wishes to bring forward the window in the lowerleft corner, which is currently underneath two other windows. Withoutgaze vector fusion there are two means to do this, and both involvemovement of the hand to another position. The first means is to move themouse pointer over the window of interest and click the mouse button.The second means is to use a hot-key combination to cycle through thescreen windows until the one of interest is brought forward. Voice inputcould also be used but it would be less efficient than the other means.With gaze vector fusion, the task is greatly simplified. For example,the user directs his gaze to the window of interest and then taps aspecific chord on the multi-touch surface. The operation requires notranslation of the hands and is very fast to perform.

For another example, assume the user wishes to resize and reposition aniTunes window positioned in the upper left of a display screen. This canbe accomplished using a multi-touch system by moving the mouse pointerinto the iTunes window and executing a resize and reposition gesture.While this means is already an improvement over using just a mouse itsefficiency can be further improved by the temporal fusion of gaze vectordata.

Device Dynamics Fusion

Device dynamics include the forces acting on a mobile or hand-held heldelectronic device that result in translations in space and rotationsabout the device's principal axes. Rotations about the principal axescan be described as roll, yaw, and pitch while the translation of thedevice can be considered relative to the body of the user.

One can envision a number of applications for mobile or hand-heldelectronic devices that would benefit from the temporal fusion of devicedynamics with multi-touch movement data. These applications wouldinclude, but not limited to, those that require a high level ofinteraction between the user and the application, as with, for example,games. Non-game applications could also benefit from the fusion ofdevice dynamics with multi-touch movement data. For example, whiletrying to use a multi-touch user interface under highly dynamicconditions such as found when riding in an airplane or walking. In caseslike these the fusion of device dynamics with multi-touch movement datacould result in a smoothing out (i.e., improved filtering) of unintendedfinger motion due to the means of traveling (e.g., vibrations andjolts).

As another example, we cite an application that involves the temporalfusion of the position of the device relative to the user's body withmulti-touch sensor data. The example application is a multi-touch cellphone with its user-side surface almost completely taken up by amulti-touch sensor. The task is to determine whether the cell phone isrelative to the user's body in order to enable the appropriate devicefunction. For example, when the phone is far away from the user's earand being held in a facing up position the multi-touch sensor inputwould be interpreted as finger movement data which is used to controlsome aspect of the device, for example, volume or selection from a list.In other positions, say, when the device is near the ear the multi-touchsensor input would be interpreted as image data and used to disablemovement control. Alternatively, the image data of the device near theear could be used to adjust some device aspect such as output volume,which could be changed depending on how close the ear is to themulti-touch surface.

Biometrics Fusion

Biometric inputs include, but are limited to, hand size, fingerprintinput, body temperature, heart rate, skin impedance, and pupil size.Typical applications that might benefit from the fusion of biometricdata with multi-touch movement data would include games, security, andfitness related activities.

Hand characteristics such as size, shape, and general morphology can beused to identify an individual for the purpose of allowing access tosecured areas, including computer systems. While hand characteristicsalone would not provide a sufficient level of identity verification, itcould be the first door through which a user must pass before othersecurity measures are applied. The fusion of physical handcharacteristics with multi-touch movement data (e.g., a trajectorysignature) would offer benefits to the initial screening process oftypical security systems.

Facial Expression Fusion

As with speech, there is much research on the machine analysis andinterpretation of facial expressions. Like the other sensing modalities,embodiments of the invention propose the fusion of facial expressiondata with multi-touch movement data and not the analytical methods used.Facial expressions convey emotional state that could be fusedadvantageously with multi-touch movement data during creative activitiessuch as music composition. Other activities such as the detection ofimpending problems via facial recognition could be fused withmulti-touch data to correct the course of events as described in thefollowing example.

A simple example of the benefit derived from the fusion of emotionalstate, as identified by facial expression, with multi-touch movementdata illustrates the possible utility of this method. As with allcomputer systems, and many electronic devices, there exists, especiallyfor novice users, a certain level of frustration when attempting tocomplete some operation. The source of the frustration can be attributedto faults in the application, ignorance of the user, or both. When thefrustration is due only to the user's ignorance he would typically beunaware of his role in causing the problem. Rather than lay blame onmachine or himself, the typical user would mostly be interested inresolving the problem and moving on to complete the task he set out todo. Remedial action by the machine could be initiated automatically ifonly the machine understood the emotional state of the user.

As discussed above, a novice user may experience frustration from timeto time when learning how to perform some task with an electronicdevice. For example, let's say that the user is trying to scroll througha document using a two-finger vertical movement (gesture). Scrolling,however, is not working for him because he is unknowingly touching thesurface with three fingers instead of the required two. He becomesfrustrated with the “failure” of the device. However, in this case, thesystem recognizes the frustration and upon analyzing the multi-touchmovement data concludes he is trying to scroll with three fingers. Atthis point, the device could bring the extra-finger problem to theattention of the user or it could decide to ignore the extra finger andcommence scrolling. Subsequent emotional data via facial recognitionwould confirm to the system that the correct remedial action was taken.

It should be appreciated that these embodiment/examples are given by wayof example and not by way of limitation. Other sensing modalities can beutilized as for example force sensing. Force sensing could be used tohelp interpret hard and light touches so as to discount or filter out adropped finger that is not part of a chord or gesture. Because thedropped finger is simply that, it does not apply as much force as thefinger implementing the gesture. Force sensing could also be used in 3Dmodeling applications to adjust the Z position of some object.

It should also be noted that other examples for each of the fusionembodiments mentioned above can be contemplated. For example, gazevector may be used to select a displayed object while MT sensing may beused to enter a command that modifies the displayed object. In oneexample, the computer may provide a grouping of photos. The user maygaze at a particular photo and perform a find gesture that causes theapplication to look for photos with similar characteristics orattributes.

It should further be noted that embodiments of the invention are notlimited to only one fused sensing modality and that multiple sensingmodalities may be used. For example, gaze sensing and voice input may beused to supplement MT data during an inputting sequence. For example,while manipulating a first object with MT data, a user can look atanother object and say green to effect a color change of the secondobject.

Embodiments of the invention are directed at improvements to theoperation and use of touch-sensitive devices such as single touch orMulti-touch (MT) devices. MT devices are configured to recognizemultiple points of contact on a near a surface at the same time. Thearrangement of contacts, which are sometimes referred to as chords, andthe motions thereof, which are sometimes referred to as gestures, can beused to generate a large number inputs including for example static andmanipulative commands.

MT devices have advantages over conventional single point sensing touchdevices in that they can distinguish more than one object (finger).Single point devices are simply incapable of distinguishing multipleobjects. In most cases, MT devices monitor a sensing surface for a touchor near touch, and when a touch occurs determines the distinct areas ofcontact and identifies the contacts via their geometric features andgeometric arrangement. Once identified or classified, the contacts aremonitored for various motions, actions or events. The contacts andmotions thereof are then converted into inputs for controlling someaspect of an electronic device.

MT devices can be embodied in various forms including but not limit tostandard touch pads, large extended palm pads, touch screens, touchsensitive housings, etc. Furthermore, MT devices can be placed invarious electronic devices including but not limited to computers suchas tablet computers, laptop computers, desktop computers as well ashandheld computing devices such as media players (e.g., music, video,games), PDAs, cell phones, cameras, remote controls, and/or the like.The MT devices may also be placed on dedicated input devices such astouch screen monitors, keyboards, navigation pads, tablets, mice, andthe like. Essentially, MT devices can be applied to any surface, and maybe found in any consumer electronic product that requires inputs.

Because MT devices provides a plethora of inputting operations at asingle location (input surface), inputting with MT devices can be veryefficient. The user can maintain their hand(s) at the MT surface withouthaving to move their hand(s) to address other input devices. Forexample, conventional systems typically include a keyboard and aseparate mouse. In order to use the mouse, the user must move their handfrom the keyboard and onto the mouse. In order to keyboard efficiently(both hands), the user must move their hand from the mouse to thekeyboard. This inputting sequence is very inefficient. For one, only onedevice can be used effectively at a given time. For another, there iswasted time between each inputting step. In contrast, with MT surfacesthe user can generate both static commands (e.g., keyboarding) andmanipulative commands (e.g., tracking) from the same location and at thesame time. The user therefore does not have to move their hands toperform different inputting tasks. The user simply provides differentchords or finger motions to generate a plethora of inputs eithersequentially or simultaneously. In one example, the user may provide keycommands with taps at specific locations of the MT surface whileallowing tracking from all locations of the MT surface.

Although input efficiency is greatly enhanced with MT devices, MTdevices still have some limitations. For one, MT sensing may producedata that is ambiguous or unclear. For example, while it may be great atdetermining the number of contacts, it may have a difficult timeascertaining the exact identity of the contact (e.g., which finger).This is especially true when there are only a limited number of contactsbeing detected. For another, in MT operations, there are typically amaximum number inputs based on various chords, and finger motions.

Therefore, in accordance with one embodiment, the invention proposesutilizing secondary sensing or input mechanisms or systems to helpinterpret the MT data. In so doing, the inputs associated therewith canbe improved and even expanded. For example, by clearly identifying eachcontact in an arrangement of contacts, more chords and gestures can becreated. The input language is no longer limited to number of contacts,and can be expanded to include a specific fingers or arrangement offingers (e.g., thumb, index, ring, middle, pinky, palm, etc.).

In addition, in some situations, the MT sensing data may not be asprecise as it should or needs to be top operate flawlessly. For example,inaccurate or course recognition of contacts and movements may lead toundesirable results. By way of example, the user's action may beinterpreted as something that it is not. The user may become annoyed ornot trust the device and as a result stop using the device. Therefore,secondary sensing devices may be used to correct, filter, smooth orotherwise positively improve the MT data so as to enhance theperformance of the inputs provided by the MT device.

In accordance with another embodiment, the invention proposes utilizingsecondary sensing or input mechanisms or systems to supplement the MToperations so as to improve the overall MT inputting experience. Forexample, the number of inputs may be further increased. As mentioned, MTdevices allow a user to implement a plethora of inputs. In order to dothis, however, the user typically maintains their hands near or at theMT surface. Moving a hand(s) away from the surface reduces the number ofavailable inputs and thus the efficiency. Thus, secondary sensing orinput mechanism or systems that allow the hand(s) to stay near the MTsurface may be used. For example, mechanisms or systems either (1)capable of sensing proximate the MT surface and/or (2) capable ofsensing something other than a hand (hands free inputting). Examples ofthe former may for example include force sensing, image sensing, opticalsensing, position sensing, motion sensing, biometric sensing and/or likeat the MT surface. Examples of the later include voice recognitionsystems, gaze vector systems, biometric systems, device dynamicssensing, environmental sensing, and/or the like.

Additional details on implementations of touch devices including MTdevices and operational methods thereof are provided in: (1), U.S.patent application Ser. No. 10/654,108 filed Sep. 2, 2003, entitled“AMBIDEXTROUS MOUSE”; (2) U.S. patent application Ser. No. 10/789,676filed Feb. 27, 2004, entitled “SHAPE DETECTING INPUT DEVICE; (3) U.S.patent application Ser. No. 10/840,862 filed May 6, 2004, entitled“MULTIPOINT TOUCHSCREEN”; (4) U.S. patent application Ser. No.11/115,539 filed Apr. 26, 2005, entitled “HAND HELD ELECTRONIC DEVICEWITH MULTIPLE TOUCH SENSING DEVICES”; (5) U.S. patent application Ser.No. 11/241,839 filed Jul. 30, 2004, entitled “PROXIMITY DETECTOR INHANDHELD DEVICE”; (6) U.S. Provisional Patent Application No. 60/658,777filed Mar. 4, 2005 entitled “MULTI-FUNCTIONAL HAND-HELD DEVICE”; (7)U.S. patent application Ser. No. 10/903,964 filed Jul. 30, 2004,entitled “GESTURES FOR TOUCH SENSITIVE INPUT DEVICES”; (8) U.S. patentapplication Ser. No. 11/038,590 filed Jan. 18, 2005 entitled “MODE-BASEDGRAPHICAL USER INTERFACES FOR TOUCH SENSITIVE INPUT DEVICES”; (9) U.S.patent application Ser. No. 11/048,264 filed Jan. 31, 2005 entitled“GESTURES FOR TOUCH SENSITIVE INPUT DEVICES”; (10) U.S. patentapplication Ser. No. 11/228,737 filed Sep. 16, 2005 entitled “ACTIVATINGVIRTUAL KEYS OF A TOUCH-SCREEN VIRTUAL KEYBOARD”; (11) U.S. patentapplication Ser. No. 11/228,758 filed Sep. 16, 2005 entitled “VIRTUALINPUT DEVICE PLACEMENT ON A TOUCH SCREEN USER INTERFACE”; (12) U.S.patent application Ser. No. 11/228,700 filed Sep. 16, 2005 entitled“OPERATION OF A COMPUTER WITH TOUCH SCREEN INTERFACE”; (13) U.S. patentapplication Ser. No. 10/927,925 filed Aug. 26, 2004 entitled “VISUALEXPANDER”; (14) U.S. patent application Ser. No. 10/927,575 filed Aug.25, 2004 entitled “WIDE TOUCHPAD ON A PORTABLE COMPUTER”, (15) U.S.patent application Ser. No. 11/015,434, filed on Dec. 17, 2004, entitled“METHOD AND APPARATUS FOR INTEGRATING MANUAL INPUT,” (16) U.S. Pat. No.6,323,846, (17) Provisional U.S. Patent Application No. 60/072,509 filedJan. 26, 1998, (18) Provisional U.S. Patent Application No. 60/763,605filed Jan. 30, 2006, entitled GESTURING WITH A MULTIPOINT SENSING DEVICE(19) U.S. patent application Ser. No. 11/057,050, filed on Feb. 11,2005, entitled “DISPLAY ACTUATOR,” (20) U.S. Pat. No. 6,677,932, (21)U.S. Pat. No. 6,570,557, (20) U.S. Pat. No. 7,030,861, (22) U.S. Pat.No. 6,888,536, all of which are herein incorporated by reference.

These and other aspects of the embodiments of the invention arediscussed below with reference to FIGS. 1-25. However, those skilled inthe art will readily appreciate that the detailed description givenherein with respect to these figures is for explanatory purposes as theinvention extends beyond these limited embodiments.

FIG. 1 is a block diagram of an electronic device or system 50, inaccordance with one embodiment of the present invention. The electronicdevice 50 may correspond to a computer, such as a desktops, laptops,tablets or handheld computers. The electronic device may also correspondto other handheld computing devices, such as cell phones, PDA, mediaplayers, media storage device, camera, remote control, and/or the like.The electronic device may also be a multifunctional device that combinestwo or more of these device functionalities into a single device.Examples of multifunctional devices can be found in U.S. ProvisionalPatent Application No. 60/658,777 filed Mar. 4, 2005 and entitled“MULTI-FUNCTIONAL HAND-HELD DEVICE”, which is herein incorporated byreference.

The exemplary electronic device 50 includes a processor 56 configured toexecute instructions and to carry out operations associated with theelectronic device 50. For example, using instructions retrieved forexample from memory, the processor 56 may control the reception andmanipulation of input and output data between components of theelectronic device. The processor 56 can be implemented on a single-chip,multiple chips or multiple electrical components. For example, variousarchitectures can be used for the processor 56, including dedicated orembedded processor, single purpose processor, controller, ASIC, and soforth. By way of example, the processor may include microprocessors,DSP, A/D converters, D/A converters, compression, decompression, etc.

In most cases, the processor 56 together with an operating systemoperates to execute computer code and produce and use data. Operatingsystems are generally well known and will not be described in greaterdetail. By way of example, the operating system may correspond to OSX,OS/2, DOS, Unix, Linux, Palm OS, and the like. The operating system canalso be a special purpose operating system, such as may be used forlimited purpose appliance-type computing devices. The operating system,other computer code and data may reside within a memory block 58 that isoperatively coupled to the processor 56. Memory block 58 generallyprovides a place to store computer code and data that are used by thecomputer system 50. By way of example, the memory block 58 may includeRead-Only Memory (ROM), Random-Access Memory (RAM), flash memory, harddisk drive and/or the like. The information could also reside on aremovable storage medium and loaded or installed onto the computersystem 50 when needed. Removable storage mediums include, for example,CD-ROM, PC-CARD, memory card, floppy disk, magnetic tape, and a networkcomponent.

The electronic device 50 also includes a display device 68 that isoperatively coupled to the processor 56. The display device 68 may be aliquid crystal display (LCD). Alternatively, the display device 68 maybe a monitor such as a monochrome display, color graphics adapter (CGA)display, enhanced graphics adapter (EGA) display,variable-graphics-array (VGA) display, super VGA display, cathode raytube (CRT), and the like. The display device may also correspond to aplasma display, a display implemented with electronic inks, or anorganic light emitting diode (OLED) display. The display device 68 maybe integrated with the electronic device 50 or it may be separatecomponents (e.g., peripheral devices). In some cases, the display device68 may be connected to the electronic device 50 through wiredconnections (e.g., wires/cables/ports). In other cases, the displaydevice 68 may be connected to the electronic device 50 through wirelessconnections. By way of example, the data link may correspond to PS/2,USB, IR, RF, Bluetooth or the like (among others).

The display device 68 is generally configured to display a graphicaluser interface (GUI) 69 that provides an easy to use interface between auser of the computer system and the operating system or applicationrunning thereon. Generally speaking, the GUI 69 represents programs,files and operational options with graphical images. The graphicalimages may include windows, fields, dialog boxes, menus, icons, buttons,cursors, scroll bars, etc. Such images may be arranged in predefinedlayouts, or may be created dynamically to serve the specific actionsbeing taken by a user. During operation, the user can select andactivate various graphical images in order to initiate functions andtasks associated therewith. By way of example, a user may select abutton that opens, closes, minimizes, or maximizes a window, or an iconthat launches a particular program. The GUI 69 can additionally oralternatively display information, such as non interactive text andgraphics, for the user on the display device 68.

The electronic device 50 also includes an input arrangement 70 that isoperatively coupled to the processor 56. The input arrangement 70 isconfigured to transfer data from the outside world into the electronicdevice 50. The input arrangement 70 may for example be used to performtracking and to make selections with respect to the GUI 69 on thedisplay 68. The input arrangement 70 may also be used to issue commandsin the electronic device 50. The input arrangement 70 may be integratedwith the electronic device 50 or they may be separate components (e.g.,peripheral devices). In some cases, the input arrangement 70 may beconnected to the electronic device 50 through wired connections (e.g.,wires/cables/ports). In other cases, the input arrangement 70 may beconnected to the electronic device 50 through wireless connections. Byway of example, the data link may correspond to PS/2, USB, IR, RF,Bluetooth or the like (among others).

In accordance with one embodiment, the input arrangement 70 includes atleast a MT data fusion inputting system. Multi-touch Data Fusion is theconcept of uniting, merging or blending MT sensing with other sensingmodalities to create a new approach to inputting. It is generallyaccomplished in a synergistic manner (cooperative action of two or moreactions). MT data fusion may be is defined as the combination of datafrom one or more independent sensing or input modalities with MT datafrom a MT device in order to improve the operation and use of anelectronic device. As shown, the MT data fusion inputting systemincludes a MT device 72 and one or more MT data fusion devices 74.

The MT device 72 is configured to receive input from a user's touch andto send this information to the processor 56. By way of example, the MTdevice 72 may correspond to a touchpad, a touch screen, a touchsensitive housing or other related touch device. As mentioned, MTsensing is capable of distinguishing multiple touches that occur at thesame time. Generally, the MT device 72 recognizes touches, as well asthe position and magnitude of touches on a multi touch sensitivesurface. The MT device 72 reports the touches to the processor 56 andthe processor 56 interprets the touches in accordance with itsprogramming. For example, the processor 56 may initiate a task inaccordance with a particular touch or touch event. The processor mayinclude a set of instructions that recognizes the occurrence of chordsand movements thereof and informs one or more software agents of whataction(s) to take in response to the chords and movement. A dedicatedprocessor can be used to process touches locally and reduce demand forthe main processor of the electronic device.

The MT data fusion devices 74 are configured to provide secondaryinformation or data that can be fused with the MT data in order tosignificantly enhance and improve input operations for the user. Thesecondary may be provided by a wide variety of sources. By way ofexample, the sources may include a variety of sensors including but notlimited to biometric sensors, audio sensors, optical sensors, sonarsensors, vibration sensors, motion sensors, location sensors, lightsensors, image sensors, acoustic sensors, electric field sensors, shocksensors, environmental sensors, orientation sensors, pressure sensors,force sensors, temperature sensors, and/or the like.

The sensors may be located at a wide variety of locations relative tothe MT surface. In some cases located above, in other cases locatedbelow. The sensors may even be level with the MT surface. The sensorsmay be located around the periphery of the MT surface or they may befound within the plane of the MT surface. In one example, pixilated orlocalized sensors are embedded in or layered with the MT surface. Forexample, temperature sensitive thin film resistors may be spreadthroughout the panel I order to provide temperature data at localizedareas of the MT surface.

In specific examples, the one or more MT data fusion devices may beselected from voice recognition, image recognition, gaze recognition,mood recognition, biometric, environmental, device dynamics and/or thelike.

The fusing may be accomplished with the processor or alternatively aseparate dedicated processor that is part of the MT data fusioninputting system. When and where the data is fused can be widely varied.Integration may for example come early, late or at the same time.

Generally, the MT data fusion device performs some operation such assensing and may include processing components for processing any data.The MT data fusion device reports secondary data to the processor 56 andthe processor 56 interprets the secondary data in context of the MT datain accordance with its programming. In one embodiment, the secondarydata may be used to help interpret MT data. Additionally oralternatively, the secondary data may be used to supplement the MT datain order to streamline input operations. For example, the secondary datamay enable multitasking, chaining and continuous stream inputting.Additionally or alternatively, the secondary data may be used to createa new input altogether. That is, MT data+secondary data=command.

The processor may have a translator that receives both sensing data andrefers to a database to determine the correct course of action. Thelanguage map may include a set of instructions that recognizes both anarrangement of contacts on MT surface and other sensing data, recognizesthe occurrence of the events and informs one or more software agents ofthe events and/or what action to take in response to the events.

In one embodiment, the secondary data and MT data is fused temporally.Generally, temporal fusion signifies an apparent coming together ofseveral events in time (multiple sets of data) within a singleindividual context (computer application, mode or platform). Morespecifically, in temporal fusion, the data from multiple sources(MT+other sensing modality) is interpreted in a time coherent manner.The information flowing from the various sources are combined or fusedsuch that events in each data stream are time aligned with each other.As such, the multiple data streams can be properly understood inconjunction with the other.

FIG. 2 is a Multi-touch (MT) data fusion method 100, in accordance withone embodiment of the present invention. The method 100 include block102 where MT sensing operations are performed. The method 100 alsoincludes block 104 where secondary sensing operations are performed. Themethod 100 also includes block 106 where the sensing data is temporallyfused with the MT data.

FIG. 3 is a Multi-touch (MT) data fusion method 120, in accordance withone embodiment of the present invention. The method 120 include block122 where an MT data stream is generated. The method 120 also includesblock 124 where one or more secondary data streams are generated. Thesecondary data stream is included to supplement or interpret the MTdata. The method 120 also includes block 126 where events in each datastream are time aligned so that MT events are properly understood inconjunction with the secondary events.

FIG. 4 is a method of inputting 140, in accordance with one embodimentof the present invention. The method 140 includes block 142 where MTsensing data is collected. The method 14 o also includes block 144 wheresecondary sensing data is collected. The method 140 additionallyincludes block 146 where in a time coherent manner events of sensingdata are combined with events of MT data to initiate an action.

FIG. 5 is a method of inputting 160, in accordance with one embodimentof the present invention. The method 160 includes block 162 where MTsensing data is collected. The method 160 also includes block 164 wheresecondary sensing data is collected. The method 160 additionallyincludes block 166 where in a time coherent manner the sensing data isutilized to supplement the MT data (simultaneous or sequential) suchthat an inputting session is fluid, seamless and efficient.

FIG. 6 is a method of inputting 180, in accordance with one embodimentof the present invention. The method 180 includes block 182 where MTsensing data is collected. The method 180 also includes block 184 wheresecondary sensing data is collected. Although shown as consecutiveblocks, it should be noted that the blocks are typically performed inthe same time domain so events in one correspond to events in the other.The method 180 additionally includes block 186 where in a time coherentmanner the sensing data is utilized to interpret the MT data so that MToperations can be expanded and improved. By way of example, thesecondary data may be used to make assumptions about the MT data, toanticipate MT data, to make provisions for MT data, to correct MT data,to filter MT data, etc. In one embodiment, the secondary data is used tobetter determine or define characteristics, attributes or behaviorsassociated with individual contacts, specific arrangement of contacts,motions of one or more contacts, motions of contacts relative to eachother, etc. By way of example, the secondary data may provide additionalcontacts information regarding Z magnitude or pressure, x coordinate, ycoordinate, ID, angle, area. The secondary data may also provideinformation regarding chord grouping, distance between contacts,centroid of contact arrangement, pressure of contact arrangement,rotation or translation of contacts relative to one another or as agroup of contacts, speed of individual contacts, speed of arrangement ofcontacts, relative speed between contacts, etc.

FIG. 7 is a method of inputting 200, in accordance with one embodimentof the present invention. The method 200 includes block 202 where anapplication, system or platform is active. The method also includesblock 204 where MT sensing is performed, and chords and movement dataare recognized. The method 200 also include block 206 where actionsassociated with the chording and movement data are initiated. The method200 also includes block 208 where voice recognition is performed, andvoice patterns are recognized. The method 200 also includes block 210where actions associated with the voice patterns are initiated.

FIG. 8 is a method of inputting 210, in accordance with one embodimentof the present invention. The method begins at block 212 where adisplayed object is selected or created during MT input. If no object isselected or created, then the method 210 waits. If an object is selectedor created, the method 210 proceeds down two simultaneously occurringblocks 214 and 216. In block 214, a determination is made as to whetheror not a voice modify command is recognized while object is selected orcreated. If so, the method 210 continues to block 218 where the selectedor created object is modified in accordance with command. If not, themethod 210 flows back to block 212. In block 216, a determination ismade as to whether or not a MT command is recognized while object isselected or created. If so, the method 210 continues to block 220 wherethe selected or created object is modified in accordance with command.If not, the method flows back to block 212.

FIGS. 9A-9E are diagrammatic illustrations of fusing voice data with MTdata according to embodiments of the invention. As shown in FIG. 9A, auser grabs a displayed object 230 with multiple fingers 232 and 234.This essentially selects the displayed object 230 for adjustments orchanges. As shown in FIG. 9B, the user rotates fingers 232 and 234 inorder to rotate the displayed object 230 (e.g., the object rotatessimultaneously with the rotation of the fingers). As shown in FIG. 9C,the user slides their fingers 232 and 234 together across the displayedsurface in order to move the displayed object 230 from one location to anew location in the viewing area or on the displayed page. As shown inFIGS. 9D and 9E, the user resizes the displayed object by pinching theirfingers 232 and 234 or spreading their fingers 232 and 234 apart.

At any time during these operations, the user can further modify thedisplayed object 230 with voice commands. For example, the user maysimply call out a color in order to change the color of the object.Alternatively, the user may state “insert text” and then speechthereafter is inputted within the displayed object 230. As should beappreciated, voice commands enable the user to continue MT inputtingwithout interruption. Additionally or alternatively, the user may usegaze recognition to further enhance the inputting operation. Forexample, the user may gaze at another object and subsequently look at anew location to move a second object 231 to that location. Again thiscan be performed while the user is performing MT inputting on the firstobject 230. As a result, inputting can be more efficient.

FIG. 10 is another diagram showing the fusion of voice recognition andMT operations according to embodiments of the invention. In thisembodiment, the user is creating a line segment 240 by dragging twofingers 234 and 235 across the screen. At any point during thiscreating, the user can call out voice commands to modify the linesegment 240. For example, the may call out “green” to change the linecolor of the line segment 240 to green. They may also call out “5 point”to change the thickness of the line segment 240. The user may also callout “dash” to change the line segment 240 to a dashed line. In all ofthese examples, the user can continue with MT inputting thereby makinginputting more efficient.

It should be noted that embodiments of the invention are not limited tocolor change and insert text commands, and that other commands may beused (e.g., open, close, etc.).

FIG. 11 is a method 250 for unambiguous finger identification, inaccordance with one embodiment of the present invention. The method 250includes block 252 where a hand profile is generated via a first sensingmechanism during inputting task. The hand profile indicating specificfingers of the hand and their location within a plane. The location maybe specific to a single axis or to multiple axis as for example the Xand Y axis of the plane. The hand profile may be created from below,level with or above the plane. The hand profile may for example begenerated via an imaging device such as an image sensor. The method 250also includes block 254 where a contact profile is generated via asecond sensing mechanism during the inputting task. The contact profileindicating one or more contact regions caused by fingers touching asurface and their location within the plane. The location may bespecific to a single axis or to multiple axis as for example the X and Yaxis of the plane. If a single axis is used, then it at leastcorresponds to the axis used for the hand profile. In most cases, theaxis that transverses or is substantially perpendicular to the fingersis used. The contact profile may be generated using a MT device thatsenses touches at a surface of the plane. The method 250 furtherincludes block 256 where the identity of the contacts are determined bycorrelating or matching the hand profile with the contact profile. Thismay be accomplished by comparing the location of the fingers with thelocation of the contacts. For example, the hand profile may indicatefingers at specific X locations, and the contact profile may indicatecontacts at specific X locations. The contact location that best fitsthe finger location may be identified as that finger. Essentially, thereare two data sets, and you look at where they coincide. Various methodsmay be implemented including match filtering and spatial correlation.

FIG. 12 is a method 270 for unambiguous finger identification, inaccordance with one embodiment of the present invention. The method 270includes block 272 where an image of the hand including fingers locatedabove MT surface is obtained. The method 270 also includes block 274where the hand image is analyzed to determine identity of fingers. Themethod 270 also includes block 276 where image of fingers in contactwith MT surface are obtained. The method 270 additionally includes block278 where the hand image and contact image are correlated to determineidentity of contacts. Although the term “image” is used and shown itshould be noted that the data may come in other forms (e.g., signals orother mathematical functions).

FIG. 13 is an inputting method 280, in accordance with one embodiment ofthe present invention. The inputting method 280 begins at block 282where a determination is made as to whether or not a touch is detected.If a touch is detected, the method 280 proceeds to blocks 284 and 286.In block 284, the hand(s) positioned over the MT surface are imaged withan image sensor. The image sensor may be embedded in the MT surface, atthe periphery of or around the edges of the MT surface, or at a locationabove the MT surface. In block 286, the fingers in contact with the MTsurface are imaged. Following both blocks 284 and 286 are blocks 288 and290. In block 288, the hand image is analyzed to determine fingeridentity and their location within the MT surface plane. In block 290,the contact image is analyzed to determine contact location within theMT surface plane. Thereafter, in block 292, the contacts are matched upwith the fingers so that the identities of the contacts are known.Although the term “image” is used and shown it should be noted that thedata may come in other forms (e.g., signals or other mathematicalfunctions).

FIG. 14A is an illustration of an exemplary image 300 of a pair of handsthat are positioned over a MT surface, and FIG. 14B is an illustrationof an exemplary image 302 of an arrangement of contacts at the MTsurface. As shown in 14A, the image 300 of the hands is processed toidentify the fingers and determine the X location of each of thefingers. The image may provide a modulated pattern of bumps in the Xcoordinate. The image system may detect the interstitial spaces betweenbumps to determine which finger is creating the contact. The imagesystem may examine the hue of the fingers, examine contrast betweenfingers (bright/light spots), examine size of each finger, or look forfinger nails, and/or the like. The thumbnails may be easily identifiablerelative to the other fingers since the thumb is typically pronatedrelative to the other fingers (thus they have a different shape). Asshown in 14B, the image 302 of the contacts is processed to determinethe X location of each of the contacts. Once the X locations are found,they are compared to the X locations of the fingers to determine theidentity of the contacts. It should be noted that this will typicallynot be an exact fit and thus a best fit algorithm may be applied to helpdetermine which contacts are the likely matches for the fingers. In theillustrated example, a first contact is located at X1, which correspondsto the left pinky at X1, the second contact is located at X2, whichcorresponds to the left ring finger, and a third contact is located atX7, which corresponds to the right index finger. Although the term“image” is used and shown it should be noted that the data may come inother forms (e.g., signals or other mathematical functions).

The hand profiles may be used to create dynamic tractor templates. Inone case, the minimum sum of distance squared may be use to identifywhich contact goes with which finger.

FIGS. 15A and 15B are diagrams of an electronic device 310, inaccordance with one embodiment of the present invention. The electronicdevice 310 includes a MT surface 312 for obtaining contact informationassociated with fingers in contact with the MT surface 312, and animaging device 314 for obtaining images of hand(s) hovering over the MTsurface 312. As shown, the image sensor 314 is positioned above thehands relative to the plane and at a top edge of the MT surface 312. Itis configured to create an image of the hand(s) relative to the X axis.Furthermore, the image sensor 314 is embedded underneath or recessedbehind the top surface of the electronic device 310. It is configured toimage the hand through an optically member 316 disposed at the topsurface of the electronic device 310. In some cases, the optical member316 may be configured to hide the image sensor 314 from view as forexample behind a bezel or opaque portion. In one embodiment, the imagesensor 314 is angled relative to the top surface (as shown) so that itcan image the hands but be hidden underneath a bezel portion 318. Forexample, it may be angled 30 degrees relative to top surface.

FIG. 16 is a diagram of an electronic device 320, in accordance withanother embodiment of the present invention. In this embodiment, theelectronic device 320 is a laptop computer having a base 322 and apivotal lid 324. The base 322 includes a MT surface 326 while the lid324 includes a display 328 that is surrounded by a bezel 330. Locatedwithin the bezel 330 is an image sensor 332 that is located high abovethe MT surface 326 and that is capable of imaging the MT surface 326. Assuch, hand profiles may be collected. Because of the angle of the lid324 relative to the base 322 changes during use, the image sensor 332may include a servo for tilting the image sensor 332 in the direction ofthe MT surface 326. In one case, a sensor that monitors the angle of thelid relative to the MT surface may also be provided. The servo can becorrectly positioned for imaging the surface by referring to themeasured angle. Alternatively, a tiltable mirror may be used instead oftilting the image sensor. The image sensor may have other functionsincluding web cam. By way of example, the image sensor may be an iSightcamera that is part of the Macbooks manufactured by Apple Computer ofCupertino, Calif.

Additional Details for pivoting an image sensor to a lid may be found inpatent application Ser. No. 10/800,166, filed Mar. 12, 2004, titledCAMERA LATCH, which is herein incorporated by reference.

As shown in FIG. 17, the image sensor 332 may also be located in adisplay or all in one computer 340 such as the iMac manufactured byApple Computer of Cupertino, Calif. In one embodiment, the image sensormay be configured to image the area below and in front of the display tofind the MT surface, and subsequently align itself as needed.

With regards to the embodiments mentioned above, although image sensorsuch as CCD or CMOS sensor is described, it should be noted that othersensing modalities may be used to image the hand. For example, laserscanners or micro radar may also be used. Furthermore, in some cases theimage sensor may be pixilated image surface which can be layered andpossibly work through the MT surface.

FIG. 18 is an inputting method 350, in accordance with one embodiment ofthe present invention. The inputting method 350 fuses gaze vector withMT sensing. The method 350 includes block 352 where the location of agaze relative to a display screen is determined. The method 350 alsoincludes block 354 where a specific arrangement of contacts and possiblymotions thereof are recognized. The method 350 additionally includesblock 356 where an action associated with the arrangement of contactsare performed at the gaze location.

FIG. 19 is an inputting method 370, in accordance with one embodiment ofthe present invention. The method 370 begins at block 372 where anobject is displayed on a display screen. The method 370 proceeds toblock 374 where a determination is made as to whether or not a gaze isdirected at the object. If not, the method 370 is placed in standby. Ifso, the method 370 proceeds to block 376 where the object is modified inaccordance with a chord or gesture associated with MT sensing.

FIG. 20 is an inputting method 380, in accordance with one embodiment ofthe present invention. The method 380 includes block 382 where dataassociated with motion of electronic device is collected. The method 380also includes block 384 where MT data is collected. The method 380additionally includes block 386 where data associated with motion ofdevice is filtered out from the MT data. This method may for example beused when the user is in motion (e.g., in a car, jogging, etc.). Inthese cases, the sensitivity of the MT panel may be decreased to avoidfalse triggering of panel areas.

Examples of motion detection and filtering of touch data may be found inU.S. patent application Ser. No. 10/997,479, filed Nov. 24, 2004, titled“MUSIC SYNCHRONIZATION ARRANGEMENT”, U.S. patent application Ser. No.10/722,948, filed Nov. 25, 2003, entitled “TOUCH PAD FOR HANDHELDDEVICE”, both of which are herein incorporated by reference.

FIG. 21 is an operational method 400, in accordance with one embodimentof the present invention. The method 400 includes block 402 where MToperations are performed at an electronic device such as a handhelddevice. The method 400 also includes block 404 where orientation and/orlocation of device relative to a user is determined. The method 400 alsoincludes block 406 where the MT data is interpreted as first data ifdevice is in first orientation and/or location. The method 400additionally includes block 408 where the MT data is interpreted assecond data if device is in second orientation and/or location. By wayof example, if the device is located away from the user as for examplewhen the user holds the device in front of them, the MT sensing may beinterpreted as chording and movement data, and if the device is locatedclose to the user the MT sensing may be interpreted as imaging data. Theimage data may be used to sense environment as well as objects. Forexample, it may be used to sense that the device is close to the face.The imaging data may be used to initiate actions. For example, if theside of the face is detected then cellular actions may be initiated.

FIG. 22 is an inputting method 420, in accordance with one embodiment ofthe present invention. The method 420 includes block 422 where handcharacteristics are determined using biometric sensor(s). The method 420also includes block 424 where a trajectory signature is recognized whenperforming MT operations. The method 420 additionally includes block 426where the user is identified based on hand characteristics andtrajectory signature.

FIG. 23 is an operational method 440, in accordance with one embodimentof the present invention. The method 440 includes block 442 whereemotional data is collected by monitoring facial expressions. The method440 also includes block 444 where MT operations are performed. Themethod 440 further includes block 446 where the emotional state of theuser is determined from the emotional data. The method 440 additionallyincludes block 448 where MT data is analyzed in accordance withemotional state. The method 440 also includes block 450 where it isdetermined if MT inputs are producing desired result. For example, ifthe emotional state indicates frustration during inputting with MTdevice, then the frustration may be due to undesirable results frominputting. The method 440 additionally includes block 452 where acorrective action is initiated if the perceived desired result is notbeing achieved. For example, if the analyzing determine that thefrustration is due to an incorrect chord or motion, then a help menu maybe displayed describing the possible user error and/or the actionassociated with the input may be adjusted.

FIG. 24 is an inputting method 460, in accordance with one embodiment ofthe present invention. The method 460 includes block 462 where MToperations are performed. The method 460 also includes block 464 whereforces at MT surface are measured during MT operations. The method 466additionally includes interpreting the MT data based on force data. Forexample, hard touches may be included in array of contacts while softtouches may be filtered out. Alternatively, force sensing may be used asa way to input. For example, light touches (or hovers) may be used tohighlight a function rather than select a function. For example, a lightouch may be used to highlight a virtual key or knob even though thefinger is not actual pushed (hard touch)

FIG. 25 is an inputting method 480, in accordance with one embodiment ofthe present invention. The method 480 includes block 482 where MTsensing is performed at a surface with first MT device. The method 480also includes block 484 where MT sensing is simultaneously performed atthe surface with a second MT device. The method 480 additionallyincludes block 486 where the results of the first and second MT devicesare compared in the time domain. The method 480 further includes block488 where the appropriate MT data to use for inputting is determined. Inone example, the two signals are averaged. In another example, one ofthe signals is selected while the other is disregarded. Other secondarydata may be used to interpret both sets of data.

The various aspects, features, embodiments or implementations of theinvention described above can be used alone or in various combinations.

Embodiments of the invention are preferably implemented by hardware,software or a combination of hardware and software. The software canalso be embodied as computer readable code on a computer readablemedium. The computer readable medium is any data storage device that canstore data which can thereafter be read by a computer system. Examplesof the computer readable medium include read-only memory, random-accessmemory, CD-ROMs, DVDs, magnetic tape, optical data storage devices, andcarrier waves. The computer readable medium can also be distributed overnetwork-coupled computer systems so that the computer readable code isstored and executed in a distributed fashion.

While embodiments of this invention have been described in terms ofseveral preferred embodiments, there are alterations, permutations, andequivalents, which fall within the scope of embodiments of thisinvention. For example, another application for combining camera with MTsensing is enabling 3D hand gestures via the camera if the user has justa part of the hand touching the MT surface at some point during thegesture. This makes the problem of hand gesture recognition via cameraeasier to solve because you know when a gesture starts or ends. Mostdemos of hand gesture recognition via a camera require you hit a “startgesture” button. If the start (or end) was just a seamless part of thegesture itself, it could make the feature more attainable on currentCPUs. In other applications, the MT surface may be expanded to cover anentire desktop or work space such that multiple users can use the sameMT surface at the same time. In cases such as these, the camera mayprovide additional information including the identity of the personsworking at the MT surface. For example, the images may be used toidentify the users and then to identify the fingers of each of theusers. It should also be noted that there are many alternative ways ofimplementing the methods and apparatuses of embodiments of the presentinvention. For example, the secondary sensing modalities may be used asmodifiers for the chords and movement data. For example, the secondarysensing modality may change input mode so that the same chord can havedifferent functionality. It is therefore intended that the followingappended claims be interpreted as including all such alterations,permutations, and equivalents as fall within the true spirit and scopeof embodiments of the present invention.

What is claimed is:
 1. A data processing system, comprising: a touchsensing device configured for receiving multiple touch inputs occurringat about the same time and generating multi-touch data; an audio sensingdevice configured for receiving sound and generating audio data; and aprocessor communicatively coupled to the touch sensing device and theaudio sensing device, the processor configured for receiving themulti-touch data and performing gesture recognition to recognize agesture, receiving the audio data and performing voice recognition torecognize a voice command, and combining both the gesture and the voicecommand to perform one or more combined gesture and voice commandoperations on an electronic device.
 2. The data processing system ofclaim 1, further comprising a display for presenting a user interface,the processor further configured for: associating the gesture with anobject being displayed; and modifying a characteristic of the displayedobject in accordance with the voice command.
 3. The data processingsystem of claim 1, wherein the one or more combined gesture and voicecommand operations include performing a static command based on thevoice command on an object being manipulated by the gesture.
 4. The dataprocessing system of claim 3, wherein the static command comprises aselection of one operational state from a plurality of possibleoperational states.
 5. The data processing system of claim 3, whereinthe static command comprises selecting a parameter of the object.
 6. Thedata processing system of claim 3, wherein the static command comprisesselecting an object manipulation mode.
 7. The data processing system ofclaim 3, wherein the static command comprises an open/close command. 8.The data processing system of claim 3, wherein the static commandcomprises a text insertion command.
 9. The data processing system ofclaim 3, wherein the one or more combined gesture and voice commandoperations include performing an object manipulation command based onthe gesture while performing the static command, the object manipulationcommand and the static command at least partially overlapping in time.10. A method for performing data processing, comprising: receivingmultiple touch inputs occurring at about the same time and generatingmulti-touch data; receiving sound and generating audio data; performinggesture recognition on the multi-touch data to recognize a gesture;performing voice recognition on the audio data to recognize a voicecommand; and combining both the gesture and the voice command to performone or more combined gesture and voice command operations on anelectronic device.
 11. The method of claim 10, further comprising:associating the gesture with an object being displayed; and modifying acharacteristic of the displayed object in accordance with the voicecommand.
 12. The method of claim 10, further comprising performing astatic command based on the voice command on an object being manipulatedby the gesture.
 13. The method of claim 12, wherein performing thestatic command comprises selecting one operational state from aplurality of possible operational states.
 14. The method of claim 12,wherein performing the static command comprises selecting a parameter ofthe object.
 15. The method of claim 12, wherein performing the staticcommand comprises selecting an object manipulation mode.
 16. The methodof claim 12, wherein performing the static command comprises performingan open/close command.
 17. The method of claim 12, wherein performingthe static command comprises inserting text.
 18. The method of claim 12,further comprising performing an object manipulation command based onthe gesture while performing the static command, the object manipulationcommand and the static command at least partially overlapping in time.19. A non-transitory computer-readable storage medium comprising programcode for performing data processing, the program code for causingperformance of a method comprising: receiving multiple touch inputsoccurring at about the same time and generating multi-touch data;receiving sound and generating audio data; performing gesturerecognition on the multi-touch data to recognize a gesture; performingvoice recognition on the audio data to recognize a voice command; andcombining both the gesture and the voice command to perform one or morecombined gesture and voice command operations on an electronic device.20. The non-transitory computer-readable storage medium of claim 19, themethod further comprising: associating the gesture with an object beingdisplayed; and modifying a characteristic of the displayed object inaccordance with the voice command.